{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CPSC 330 hw1\n",
    "\n",
    "**About this assignment:** The main purpose of this assignment is to check whether your programming knowledge is adequate to take CPSC 330. This assignment covers two python packages, numpy and pandas, that we'll be using throughout the course. For some of you, Python/numpy/pandas will be familiar; for others, it will be new. Either way, if you find this assignment very difficult then that could be a sign that you will struggle later on in the course. While CPSC 330 is a machine learning course rather than a programming course, programming will be an essential part of it.\n",
    "\n",
    "Also: as part of this assignment you will likely need to consult the documentation for various Python packages we're using. This is, of course, totally OK and in fact strongly encouraged. Reading and interpreting documentation is an important skill, and in fact is one of the skills this assignment is meant to assess.\n",
    "\n",
    "**Due date: Tuesday Sep 15, 2020 at 11:59pm.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up\n",
    "\n",
    "In order to do this assignment, you will need to set up the CPSC 330 software stack, which is Python and Jupyter. For software install help, see [here](https://github.com/UBC-CS/cpsc330/blob/master/docs/setup.md). One you have the software stack installed, you should be able to run the next cell, which imports some packages needed for the assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Points\n",
    "\n",
    "Each question or sub-question will have a number of points allocated to it. For example, the \"rubric={points:3}\" below means that following the instructions is worth 3 points/marks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "rubric={points:5}\n",
    "\n",
    "Follow the [CPSC 330 homework instructions](https://github.com/UBC-CS/cpsc330/blob/master/docs/homework_instructions.md), which includes information on how to do your assignment and how to submit your assignment. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "## Exercise 1: Loading files with Pandas\n",
    "rubric={points:6}\n",
    "\n",
    "When working with tabular data, you will typically be creating Pandas dataframes by reading data from .csv files using `pd.read_csv()`. The documentation for this function is available [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-e9d5646932eee4f9",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "In the \"data\" folder in this homework repository there are 6 different .csv files named `wine_#.csv/.txt`. Look at each of these files and use `pd.read_csv()` to load these data so that they resemble the following:\n",
    "\n",
    "| Bottle | Grape | Origin | Alcohol | pH | Colour | Aroma |\n",
    "| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n",
    "| 1 |  Chardonnay | Australia | 14.23 | 3.51 | White | Floral |\n",
    "| 2 |  Pinot Grigio | Italy | 13.20 | 3.30 | White | Fruity |\n",
    "| 3 |  Pinot Blanc | France | 13.16 | 3.16 | White | Citrus |\n",
    "| 4 |  Shiraz | Chile | 14.91 | 3.39 | Red | Berry |\n",
    "| 5 |  Malbec | Argentina | 13.83 | 3.28 | Red | Fruity |\n",
    "\n",
    "You are provided with tests that use `df.equals()` to check that all the dataframes are identical. If you're in a situation where the two dataframes look identical but `df.equals()` is returning `False`, it may be an issue of types - try checking `df.index`, `df.columns`, or `df.info()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = None\n",
    "df2 = None\n",
    "df3 = None\n",
    "df4 = None\n",
    "df5 = None\n",
    "df6 = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-464ad2febe587f69",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, df in enumerate([df2, df3, df4, df5, df6]):\n",
    "    assert df1.equals(df), f\"df1 not equal to df{i + 2}\"\n",
    "print('All tests passed.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "## Exercise 2: The Titanic dataset\n",
    "\n",
    "The file *titanic.csv* contains data of 1309 passengers who were on the Titanic's unfortunate voyage. For each passenger, the following data are recorded:\n",
    "\n",
    "* survival - Survival (0 = No; 1 = Yes)\n",
    "* class - Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)\n",
    "* name - Name\n",
    "* sex - Sex\n",
    "* age - Age\n",
    "* sibsp - Number of Siblings/Spouses Aboard\n",
    "* parch - Number of Parents/Children Aboard\n",
    "* ticket - Ticket Number\n",
    "* fare - Passenger Fare\n",
    "* cabin - Cabin\n",
    "* embarked - Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)\n",
    "* boat - Lifeboat (if survived)\n",
    "* body - Body number (if did not survive and body was recovered)\n",
    "\n",
    "In this exercise you will perform a number of wrangling operations to manipulate and extract subsets of the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-8f49a0571a8d666f",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(a)\n",
    "rubric={points:1}\n",
    "\n",
    "Load the `titanic.csv` dataset into a pandas dataframe named `titanic_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-b6e4d3bb99cced81",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "titanic_df = None\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-0abb41f189217e1a",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert set(titanic_df.columns) == set(['pclass',\n",
    " 'survived',\n",
    " 'name',\n",
    " 'sex',\n",
    " 'age',\n",
    " 'sibsp',\n",
    " 'parch',\n",
    " 'ticket',\n",
    " 'fare',\n",
    " 'cabin',\n",
    " 'embarked',\n",
    " 'boat',\n",
    " 'body',\n",
    " 'home.dest']), \"All required columns are not present\"\n",
    "assert len(titanic_df.index) == 1309, \"Wrong number of rows in dataframe\"\n",
    "print(\"Success\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-093e1ca409a216a2",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(b)\n",
    "rubric={points:1}\n",
    "\n",
    "The column names `sibsp` and `parch` are not very descriptive. Use `df.rename()` to rename these columns to `siblings_spouses` and `parents_children` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-99e694773668d8a1",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-6c0798bceeeab962",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert set(['siblings_spouses', 'parents_children']).issubset(titanic_df.columns), \"Column names were not changed properly\"\n",
    "print(\"Success\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-35f94190fc8af3fb",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(c)\n",
    "rubric={points:2}\n",
    "\n",
    "We will practice indexing different subsets of the dataframe in the following questions.\n",
    "\n",
    "Select the column `age` using single bracket notation `[]`. What type of object is returned?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-18b067281f9e13ac",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-5b123152df1d576a",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(d)\n",
    "rubric={points:2}\n",
    "\n",
    "Now select the `age` using double bracket notation `[[]]`. What type of object is returned?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-0d32262f000b09f4",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-5b9513513d0b0e22",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(e)\n",
    "rubric={points:1}\n",
    "\n",
    "Select the columns `pclass`, `survived`, and `age` using a single line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-d3fdbdd8c0af6b49",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-a7b37913a746262d",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(f)\n",
    "rubric={points:2}\n",
    "\n",
    "Use the `iloc` method to obtain the first 5 rows of the columns `name`, `sex` and `age` using a single line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-a59f3080f818f34a",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-37f2418e03a50b60",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(g)\n",
    "rubric={points:2}\n",
    "\n",
    "Now use the `loc` method to obtain the first 5 rows of the columns `name`, `sex` and `age` using a single line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-01f92751cc57fded",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-07036a85b1202239",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(h)\n",
    "rubric={points:1}\n",
    "\n",
    "How many passengers survived (`survived = 1`) the disaster? Hint: try using `df.query()` or `[]` notation to subset the dataframe and then `df.shape` to check its size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-5677d8dc540bc7ec",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-560ca2c33cf74784",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(i)\n",
    "rubric={points:1}\n",
    "\n",
    "How many passengers that survived the disaster (`survived = 1`) were over 60 years of age?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-123d66686c410308",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-64aa731d50112f88",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(j)\n",
    "rubric={points:2}\n",
    "\n",
    "What was the lowest and highest fare paid to board the titanic? Store your answers as floats in the variables `lowest` and `highest`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-54dc8e011e41a6b5",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    " \n",
    "lowest = None\n",
    "highest = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-2f06f5b42b270b44",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "#### 2(k)\n",
    "rubric={points:1}\n",
    "\n",
    "Sort the dataframe by fare paid (most to least)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2(l)\n",
    "rubric={points:1}\n",
    "\n",
    "Save the sorted dataframe to a .csv file called 'titanic_fares.csv' using `to_csv()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-4be555e930c36650",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2(m)\n",
    "rubric={points:3}\n",
    "\n",
    "Create a scatter plot of fare (y-axis) vs. age (x-axis). Make sure to follow the [guidelines on figures](https://github.com/UBC-CS/cpsc330/blob/master/docs/homework_instructions.md#figures). You are welcome to use pandas built-in plotting or matplotlib. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2(n)\n",
    "rubric={points:3}\n",
    "\n",
    "Create a histogram of `embarked` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3: Scavenger Hunt\n",
    "\n",
    "For each of the following cases, index into the Python object to obtain the \"T\" (for Treasure). Please do not modify the original line of code that generates `x` (though you are welcome to copy it). You are welcome to answer this question \"manually\" or by writing code - whatever works for you. However, your submission should always end with a line of code that prints out `'T'` at the end (because you've found it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "letters = string.ascii_uppercase"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first one is done for you as an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = (\"nothing\",{-i : l for i, l in enumerate(letters)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example answer**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'T'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[1][-19]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Explanation:** In these questions, the goal is not to understand the code itself, which may be confusing. Instead, try to probe the types of the various objects. For example `type(x)` reveals that `x` is a tuple, and `len(x)` reveals that it has two elements. Element 0 just contains \"nothing\", but element 1 contains more stuff, hence `x[1]`. Then we can again probe `type(x[1])` and see that it's a dictionary. If you `print(x[1])` you'll see that the letter \"T\" corresponds to the key -19, hence `x[1][-19]`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(a)\n",
    "rubric={points:2}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do not modify this cell\n",
    "x = [[letters[i] for i in range(26) if i % 2 == 0], [letters[i] for i in range(26) if i % 2 == 1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(b)\n",
    "rubric={points:2}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do not modify this cell\n",
    "np.random.seed(1)\n",
    "x = np.random.choice(list(set(letters)-set(\"T\")),size=(100, 26),replace=True)\n",
    "x[np.random.randint(100),np.random.randint(26)] = \"T\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(c)\n",
    "rubric={points:2}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do not modify this cell\n",
    "n = 26\n",
    "x = dict()\n",
    "for i in range(n):\n",
    "    x[string.ascii_lowercase[i]] = {string.ascii_lowercase[(j+1)%n] : [[letters[j]] if j-2==i else None] for j in range(n)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission to Canvas\n",
    "\n",
    "**PLEASE READ:** When you are ready to submit your assignment do the following:\n",
    "\n",
    "1. Run all cells in your notebook to make sure there are no errors by doing `Kernel -> Restart Kernel and Clear All Outputs` and then `Run -> Run All Cells`.\n",
    "2. Convert your notebook to `.html` format using the `convert_notebook()` function below **or** by `File -> Export Notebook As... -> Export Notebook to HTML`.\n",
    "3. Run the code `submit()` below to go through an interactive submission process to Canvas.\n",
    ">For this step, you will need a Canvas *Access Token* token. If you haven't already got one, log-in to Canvas, click `Account` (top-left of the screen), then `Settings`, then scroll down until you see the `+ New Access Token` button. Click that button, give your token any name you like and set the expiry date to Dec 31, 2020. Then click `Generate token`. Save this token in a safe place on your computer as you'll need it for all assignments. Treat the token with as much care as you would an important password. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from canvasutils.submit import submit, convert_notebook\n",
    "\n",
    "# Note: the canvasutils package should have been installed as part of your environment setup - \n",
    "# see https://github.com/UBC-CS/cpsc330/blob/master/docs/setup.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert_notebook(\"hw1.ipynb\", \"html\")  # uncomment and run when you want to try convert your notebook to HTML (or you can convert manually from the File menu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit(course_code=53561, token=False)  # uncomment and run when ready to submit "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
